From visuo-motor to language

We propose a learning agent that first learns concepts in an integrated, cross-modal manner, and then uses these as the semantics model to map language. We consider the action of throwing, considering the whole trajectory as a single image. A large set of such images, and the throwing parameters are mapped jointly onto a low-dimensional non-linear manifold. Such models improve with practice, and can be used as the starting point for real-life tasks such as aiming (e.g. dart throwing). How can such models can be used in learning language? We consider a set of videos involving throwing and rolling actions. These actions are analyzed into a set of contrastive semantic classes based on the agent, action, and the thrown object (trajector). We obtain a crowdsourced commentaries for these videos (unannotated text) from a number of adults. A learner system attempts to learn labels using the contrastive probabilities for a given semantic class. Only a handful of highconfidence words are found, but the agent starts off with this partial knowledge. These are used to learn a potential set of syntactic patterns, for example for the trajector, and then for the agenttrajector-action sentences. We demonstrate how this may work for two completely different languages English and Hindi, and also show how rudiments of agreement, synonymy and polysemy are detected.